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I.  INTRODUCTION 

A  key  concept,  elreedy  familiar  to  readers  acquainted  with  queue¬ 
ing  theory,  is  that  of  an  imbedded  Markov  chain  connecting  regeneration 
points  of  a  stochastic  process.  Starting  from  a  regeneration  point, 
the  future  is  stochastically  independent  of  the  past.  In  the  imbedded 
Markov  chain,  the  original  time  scale  of  the  transitions  between  re¬ 
generation  points  is  replaced  by  a  discrete  time  version  where  all 
transitions  take  unit  time.  The  corresponding  imbedded  semi-Markov 
process  looks  only  at  the  "states"  corresponding  to  these  regeneration 
points,  but  it  does  so  in  continuous  time. 

The  use  of  these  distinguished  states,  generally  referred  to  as 
events  by  probabilists,  is  natural  in  a  dynamic  programming  framework; 
see,  e.g.,  Denardo  and  Mitten  [13].  In  fact,  it  was  programming  over 
semi-Markov  processes  that  motivated  our  interest  in  these  processes - 
He  shall  have  more  to  say  about  this  in  Sec.  10. 

Part  of  the  definition  of  a  distinguished  state  is  its  association 
with  certain  regeneration  points.  To  fix  this  idea  concretely,  consider 
the  M/G/l  queue  (Poisson  arrivals,  general  service  time  distribution, 
single  channel).  For  many  purposes,  a  convenient  set  of  distinguished 
states  is  {0,  1,  2,  ...1  where  state  i  signifies  i  customers  in  the 
system  and  a  service  has  just  been  completed. 

This  procedure  is  definitely  in  conflict  with  the  notion  of  a 
state  used  in  fully  rigorous  treatments  of  probability  theory  and  semi- 
Markov  processes  in  particular.  There,  a  sample  path  of  a  stochastic 
process  is  defined  as  a  function  X(',  ou):  [0,  »)  -  E  and  the  members 
of  B  are  called  states.  This  often  necessitates  speaking  of  a  "holding 


H 


time"  in  a  state.  If  this  "holding  time"  is  not  necessarily  exponen¬ 
tially  distributed,  as  in  semi-Markov  processes,  then  we  intuitively 
feel  that  the  process  does  not  remain  in  the  same  state  during  the 
holding  period  because  the  process  has  memory  (is  not  Markovian).  By 
contrast,  all  our  distinguished  states  are  occupied  only  for  an  Instant. 
This  procedure  is  unconventional  and  heuristic,  but  this  is  not  sup¬ 
posed  to  be  a  review  paper  for  experts.  Rather,  it  is  directed  toward 
those  readers  with  some  prior  exposure  to  Markov  chains  and  renewal 
theory  who  would  like  to  get  a  feel  for  what  semi-Markov  processes  are 
all  about  and  how  they  arise  in  applications.  For  many  readers  belong¬ 
ing  to  this  class,  it  is  felt  that  this  primer  is  a  more  accessible  first 
introduction  than  the  original  papers  cited  in  the  reference  list. 

Letting  N^(t)  denote  the  number  of  times  state  j  is  entered  in 
the  half-open  Interval  (0,  t],  we  obtain  the  Markov  Renewal  Process 
(MRP)  N(t)  -  (Ho(t),  Nj (t) ,  N2(t),  ...).  In  the  M/0/1  queue,  for 
example,  NQ(t)  is  the  number  of  busy  periods  completed  in  (0,  t]. 

Let  Z(t),  the  semi-Markov  process,  be  the  last  distinguished  state 
entered  in  [0,  t].  In  general,  such  a  last  state  is  not  veil  defined, 
but  in  the  applications  there  is  virtually  never  any  difficulty.  See 
Sec.  3  for  discussion  of  this  point.  In  the  M/C/1  queue,  if  at  the 
last  service  completion  epoch  in  [0,  t]  there  were  1  customers  in  the 
system,  then  Z(t)  ■  i;  however,  at  time  t  there  may  be  more  than  i  cus¬ 
tomers  in  the  system  due  to  arrivals  since  the  last  service  completion. 

I+  is  defined  to  be  the  set  of  distinguished  states,  assumed 
countable.  The  state  transitions  form  a  Markov  chain  with  transition 
probabilities  (Pfj)>  where  direct  transitions  from  a  state  to  Itself 
(e.g.,  pAi  >  0)  are  allowed.  Clven  that  in  1  -  J  transition  is  about 


Co  occur,  the  duration  of  the  transition  has  distribution  F 


In  the 


ij- 

M/0/1  queue,  F, .  ■  G  If  1  >  0  and  F  ,  is  the  convolution  of  0  with  the 
ij  oj 

exponential  distribution  whose  mean  is  the  reciprocal  of  the  arrival 
rate;  the  transition  matrix  is  given  in  Sec.  7. 

Semi-Markov  processes  (SMP's),  first  studied  by  P.  Ldvy  and 
W.  L.  Smith,  generalise  several  familiar  processes.  Mote  that 
(1)  a  ona-state  SHF  is  a  renewal  process; 

(il)  an  SMF  with  F^  degenerate  at  one  for  all  i,  J  is  a  Markov  chain; 
(Hi)  an  SMF  with  all  F^  exponential  and  independent  of  j  is  a  con¬ 
tinuous  time  countable- state  Markov  process; 

(lv)  an  alternating  renewal  process  is  a  two-state  SMF. 

All  SMF' s  have  renewal  processes  imbedded  within  them  corresponding  to 
looking  only  at  successive  returns  to  the  same  state.  In  the  M/G/l 
queue,  if  the  state  in  question  is  0,  then  we  are  looking  at  successive 
returns  to  the  beginning  of  an  idle  period. 

Many  problems  in  management  science  and  operations  research  can  be 
modeled  as  SMF's:  for  example,  queueing,  inventory,  and  maintenance 
problems.  Explicit  recognition  of  the  underlying  SMF  often  streamlines 
the  analysis.  For  details,  see,  e.g.,  Fyke  [44],  Barlow  and  Froschan 
[1],  Fabens  [15],  Foley  [18],  ^inlar  [4-8],  and  Meuts  [37-41].  These 
areas  by  no  swans  exhaust  the  possibilities;  e.g.,  Perrin  and  Sheps  [43] 
and  Weiss  and  Zelen  [52]  apply  SMF's  to  medical  problems.  John  McCall 
(unpublished  work)  has  used  SMF's  to  model  movements  among  income  classes 
in  a  study  of  strategies  for  combatting  poverty.  For  proofs,  citations 
of  earlier  papers,  and  additional  topics  in  SMP's,  the  reader  should 
consult  the  reference  list.  Another  expository  paper  is  Janssen  [26]. 
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Recently,  Neuts  T42]  has  published  a  bibliography  on  SMP's. 
thank  Professor  Neuts  for  bringing  several  relevant  papers  to  my 
attention. 


-6- 


II.  DIST:  1GUISHED  STATES 

In  Chis  section  we  examine  precisely  what  is  meant  by  a  distinguished 
state.  Since  discussing  this  topic  in  an  offhand  manner  could  result 
in  confusion,  the  subject  is  treated  in  some  detail. 

For  each  sample  path  of  a  stochastic  process  X,  there  is  a 
correspondence  between  [t:  t  a  0]  and  a  set  of  states  S.  If  every 
state  (in  our  sense  of  the  word)  in  S  is  required  to  have  the  Harkov 
property,  then  in  general  S  will  be  uncountable  since  a  history  of  the 
process,  or  at  least  the  relevant  portion  of  it,  must  be  part  of  the 
state  definition.  However,  all  that  we  require  of  S  is  that  it  have  an 
appropriate  countable  subset  I*  of  (distinguished)  states  having  the 
Markov  property.  Thus,  for  a  state  to  be  a  candidate  for  I+,  it  must 
correspond  to  a  regeneration  point,  but  we  do  not  require  that  all 
states  corresponding  to  regeneration  points  belong  to  I+.  We  assume 
that  the  process  starts  in  a  distinguished  state  at  time  0. 

Note  that  the  set  of  distinguished  states  used  previously  in 
our  discussion  of  the  M/G/l  queue  does  not  include  all  regeneration 
points,  since  any  time  the  system  is  idle  (empty),  it  is  at  a  regen¬ 
eration  point.  Occasionally  it  is  convenient  to  add  to  I+  the 
state  corresponding  to  arrival  epochs  to  an  empty  system.  Our  choice 
of  distinguished  states  conforms  to  our  requirements  because  the 
time  to  the  next  arrival  is  stochastically  independent  of  the  time 
elapsed  since  the  last  arrival.  In  general,  arrival  epochs,  except 
those  corresponding  to  the  start  of  a  busy  period,  are  not  regenera¬ 
tion  points.  Thus  the  state  "i  customers  in  the  system  and  a  cus¬ 
tomer  has  Just  arrived"  cannot  be  a  distinguished  state,  unless 


1  *  0  or  the  service  times  are  exponential.  Through  the  use  of  so- 
called  supplementary  variables,  we  can  define  every  state  in  the 
original  process  so  that  it  is  Markovian.  Each  state  is  then  a  couple 
of  the  form  (i,u),  which  denotes  i  customers  in  the  system  and  the 
customer  being  processed  has  been  in  service  for  time  u;  for  1=0, 
u  is  arbitrary,  say  0.  Sometimes  supplementary  variable  techniques 
are  useful  as  an  alternative  or  adjunct  to  SMP  techniques;  see,  e.g., 

Cox  and  Miller  [9].  A  disadvantage  of  supplementary  variable  techniques 
is  that  superfluous  regularity  conditions,  e.g.,  absolutely  continuous 
transition  time  distributions,  often  must  be  imposed  on  the  original 
process  to  justify  their  use. 

Returning  to  the  general  discussion,  we  require  the  distinguished 
states  to  be  defined  such  that  non-zero  holding  times  in  a  distinguished 
state  are  forbidden1  but  Instantaneous  transitions  among  the  distinguished 
states  are  allowed.  This  is  a  departure  from  the  setup  of  Pyke  [44], 
although  the  two  formulations  are  essentially  equivalent.  The  notion 
of  an  auxiliary  path  (see  [47]) ,  needed  in  the  conventional  setup  to 
handle  such  processes  as  the  M/G/l  queue,  is  not  required  by  us.  Our 
definition  of  distinguished  state  permits  a  graphic  representation 
of  SMP's  in  terms  of  networks  with  branch  nodes  (distinguished  states) 
and  stochastic  arc  lengths;  see  [19].  For  example,  traversing  an  arc 
could  correspond  to  a  customer  completing  service. 

1By  contrast,  in  the  conventional  setup  non-zero  holding  times  . 
are  essential,  but  this  seems  to  be  an  artifice  unless  at  every 
instant  the  process  is  memoryless,  i.e.,  Markovian. 
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To  remove  ambiguity  in  case  of  instantaneous  transitions,  we 
define  X  (t)  =  X(t  );  thus,  X  is  right  continuous  and  the  last 
distinguished  state  of  X+  entered  in  [0,  t],  say,  is  well  defined, 
provided  that  the  process  does  not  explode;  see  the  discussion  of 
regularity  in  Sec.  3.  Since  we  have  prohibited  non-zero  holding 
times  in  distinguished  states,  we  cannot  allow  a  distinguished  state 
to  correspond  to  a  nondegenerate  interval  of  regeneration  points 
(e.g.,  an  idle  period  in  an  M/G/l  queue).  Thus,  to  exclude  an 
infinite  sequence  of  instantaneous  transitions  from  a  state  to 
itself,  we  require  that  the  distinguished  states  be  defined  such 
that,  for  all  nondegenerate  intervals  (a,  b) ,  i  €  I+  ^ 

P[X(t)  =  i,  Vt  €  (a,  b) }  =  0.  For  example,  in  the  M/G/l  queue  it 
does  not  suffice  to  define  the  distinguished  state  0  as  0  customers 
in  the  system.  The  condition  that  a  service  has  just  been  completed 
must  be  added. 

In  applications,  the  first  step  is  to  carefully  specify  the  dis¬ 
tinguished  states,  which  throughout  the  sequel  are  simply  called 
"states,"  necessitating  definitions  slightly  different  from  conven¬ 
tional  usage. 

2 

Sometimes  it  is  convenient  to  permit  instantaneous  transitions; 
see,  e.g.,  Denardo  [11].  Yackel  [55]  makes  a  detailed  study  of  limit 
theorems  for  SMP's  with  instantaneous  transitions  allowed. 
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III.  REGULAR  SMP'S 

In  Che  literature  the  Z  process  is  called  an  SMP.  However,  the  MRP 
and  the  SMP  are  different  aspects  of  the  same  underlying  stochastic 
process;  therefore,  by  slight  abuse  of  language,  we  shall  refer  to  the 
underlying  process  itself  as  an  SMP  or  a  MRP,  using  the  terms  inter¬ 
changeably. 

An  MRP  is  regular  if  with  probability  one  (w.p.l)  each  state  is 
entered  only  a  finite  number  of  times  in  any  finite  time  span--i.e., 
if  P[N^ (t)  <  c°  ]  =  1,  Vi  €  I+  and  t  2  0.  An  MRP  is  strongly  regular 
if  w.p.l  the  total  number  of  state  transitions  is  finite  in  any  finite 
time  span--i.e.,  if  P[£  N^(t)  <<*>]  =  1,  Vt  2  0.  Clearly  strong 
regularity  implies  regularity  and,  if  n  <  »,  it  suffices  that  H  «  (H  , . . . , 
have  at  least  one  component  nondegenerate  at  zero  for  every  ergodic 
subchain  of  the  imbedded  Markov  chain,  where  is  the  unconditional 
distribution  of  time  elapsed  starting  from  state  i  until  the  next  state 
is  entered  (possibly  i  itself).  In  the  denumerable  state  case  (n  =  «) , 
see  Pyke  [44]  and  Pyke  and  Schaufele  [46]  for  conditions  that  imply 
strong  regularity;  see  also  Feller  [17].  In  the  sequel,  we  assume  that 
strong  regularity  holds. 
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IV.  FIRST  PASSAGE  AND  COUNTING  DISTRIBUTIONS 


Let 


3 


H,(0 


-Iv‘> 

j 


Pij(t)  -  PfZ(t)  -  j|Z(0)  -  il 

G..(t)  =  PrN^t)  >  0|Z(0)  -  i] 

(first  passage  time  distribution) 


M£ j  ( t)  -  E[*  (t)|X(0)  -  i-] 

(mean  entry  counting  function). 


Defining  the  convolution 

(A  *  B) (t) 


/. 


t 

A(t  -  x)  dB(x) 


0 


and  deleting  the  argument  t  below,  we  have  by  straightforward  renewal- 
theoretic  arguments: 


pu  '  “  •  V5u  +SQiv  *  pkJ  * <l  -  Vstj  +  pjj  *  Gu 

k 

Gu  ‘  «ij +  E  ’ik  *  °k) 

"u  “ °ij +  °m  * Mij  •<>tj  *EV *kj. 

k 


3  + 

Unless  otherwise  stated,  all  summations  will  be  over  I  and  all 

functions  vanish  for  negative  arguments. 


In  general,  these  relations  cannot  be  solved  analytically,  although  for 
the  moments  complicated  expressions  have  been  obtained  (Pyke  and  Schau- 
fele  r461).  In  the  finite  state  case,  numerical  solutions  can  be  ob¬ 
tained  by  numerical  inversion  of  the  corresponding  Laplace  transforms 
(see,  e.g.,  T21,  [241,  [331,  and  [47]).  For  each  value  of  s,  only  one 
matrix  inversion  in  the  transform  domain  is  required--that  of  I  -  q(s) , 
where  s  >  0  and 


In  obvious  notation,  having  found  [I  -  q(s)]  \  either  analytically  as 
a  function  of  s  or,  for  suitably  spaced  values  of  s,  numerically,  one 
successively  computes 

m(s)  -  [I  -  q(s)]  Xq(s)  *  [I  -  q(s)]'1  -  I 

8ij(s)  ■  mij(s)/[l  +  nijj(s)l 

Pij(8)  “  1  +  3 

1  -  hj (s) 

PjJ(S>  “  1  *  gjj(s)  ’ 

and  then  inverts  the  transforms.  Although  this  procedure  is  not  trivial, 
it  often  compares  favorably  with  the  alternative  simulation  approach  for 
getting  the  transient  behavior  in  the  time  domain.  By  usual  limit  theo¬ 
rems  for  Laplace  transforms  (Widder  [531,  Feller  [161,  see  also  Jewell 
[27]),  the  behavior  in  the  time  domain  for  large  (small)  t  corresponds 
to  behavior  in  the  transform  domain  for  small  (large)  s.  Renewal  theory 
provides  an  important  tool  in  studying  asymptotic  behavior;  see  Sec.  8. 
For  the  stationary  probabilities,  see  Sec.  6. 
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Conditioning  on  the  event  that  no  state  in  a  subset  B  of  I  is 
entered  in  (0,  t]  may  be  of  interest.  For  example, 

B  ptj(t)  "  P[Z(t)  -  j|ZiO)  -  i,  Nk(t)  -  0,  Vk  €  B] 

B  G^t)  -  Pp^t)  >  0|Z(0)  -  i,  Nk(t)  -  0,  Vk  6  B] 

B  Mtj(t)  -  EfNjUJlZCO)  -  i,  Nk(t)  -  0,  Vk  €  B] 

can  be  calculated  from  the  formulas  already  given  by  (temporarily)  making 
the  states  in  B  absorbing. 

Barlow  and  Proschan  [1,  pp.  132-134],  using  ‘’renewal"  arguments, 

show  that  the  first  and  second  moments  of  ,  denoted  respectively  by 
(2) 

^  and  are  given  by 


where  ia  the  mean  of  F 


ij’ 


(1) 


v,  and 


mJ  t^dH^t). 


(2)  + 

We  assume  that  <  “i  Vi  6  I  .  If  the  imbedded  Markov  chain  is 

4 

finite  and  ergodic,  these  equations  have  a  unique  finite  solution 


and,  with  Hj  the  stationary  probability  that  the  last  state  entered 


4 

See  appendices  1  and  2  of  Fox  [19]  for  an  efficient  way  to  solve 
these  equations.  (An  expression  for  the  "bias  terms"  in  Markov  renewal 
programming  Involves  the  first  passage  time  moments,  which  are  of  intrin¬ 
sic  interest,  but  recently  Jewell  [29]  has  derived  a  remarkably  simple 
alternative  expression,  obviating  the  need  to  calculate  to  evalu¬ 
ate  the  bias  terms.)  ^ 
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5  (2) 

is  j  if  all  F.j  were  degenerate  at  one,  multiplying  ia^  and 
by  rr  and  summing  yields 


M-j j  "  (1/TTj>  L"kvk 
k 


(2)  -  (M 
M> j  j  *  (1/tt 


j'fcvk18  *  !  £  EVi^i^l 

L  k#j  i  -I 


For  finite  state  SMP’s,  the  probability  that  state  j  is  ultimately 

6 


reached  starting  from  i  is 

I. 


°ij(*) 


if  i,  J  €  E, 


1, 

0,  “  ‘  c  *V  J  ^  ~t 

[(1  -  A)"16]1,  if  i  €  T,  j  e  E 


if  i  €  Eu,  j  €  E/(  k  +  j 
k 


where  A  is  the  submatrix  of  P  corresponding  to  the  set  T  of  transient 

states,  E, ,  E  are  the  recurrent  subchains  of  P,  and 

*  1  m 


0i  *  H  Pit>  1  €  T. 
I€E,. 


The  mean  time  to  leave  T  starting  from  i  is 


-  [(I  -  A)"V]if  i  €  T  , 


where  v  is  the  vector  of 
generating  function  for 


Vj's,  j  €  T.  Pyke  [A5]  obtains  a  double 
the  distribution  of  N^(t),  viz., 


5In  other  words,  tt  is  the  stationary  measure  for  the  imbedded 
chain,  but  not  (in  general)  for  the  SMP  itself. 

^  The  case  i,  J  €  T,  of  less  interest,  is  not  considered. 
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Yz  «  1  -  (1  -  z)m  [*  I  +  (1  -  z)D]-1, 
where  1  is  a  matrix  of  l's. 


and 


\  -  %(*;  •)) 


#ij(z;  s)  mJ  e"st  dtwij(z;  t) 


OP 

wij(zj  fc)  z^vij  ^k»  O 

k-0 

v^Ck;  t)  -  PCNjCc)  -  k|Z(0)  -  i]. 


Thus,  in  principle,  the  probabilities  and  moments  can  be  obtained  in 
the  usual  way.  The  Laplace  transform  m  of  the  first  moment  (M^(*)) 
was  already  given.  See  Pyke  and  Schaufele  [46]  for  further  general 
moment  computations,  weak  and  strong  laws  of  large  numbers,  and 
central  limit  theorems.  A  generating  function  that  yields  many 
quantities  of  interest  upon  considering  special  cases  has  been  ob¬ 
tained  by  Neuts  [36].  Stone  [50]  derives  the  distribution  of  the 


maximum  of  an  SMP. 
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V.  STATE  CLASSIFICATION 


In  classifying  the  states  of  an  SMP  transient,  null  recurrent,  or 
positive  recurrent,  we  must  distinguish  between  a  state's  classifi¬ 
cation  in  the  imbedded  Markov  chain  and  in  the  SMP  itself.  For  I+ 
finite  and  <  “,  Vi  €  I+,  the  distinction  disappears  and  a  state  j 
is  either  transient  or  positive  recurrent  (Gjj(«)  *=  1  and  ^  ^  <  ®) . 

In  large-scale  applications,  the  ergodic  subchain-transient  set 
breakdown  may  not  be  obvious  and  recourse  may  be  necessary  to  an 
algorithmic  classification  scheme  such  as  that  of  Fox  and  Landi  [23]. 

For  I+  infinite,  a  state  j  is  transient  (recurrent--i. e. ,  ■  1) 

in  the  SMP  »  ]  is  transient  (recurrent)  in  the  imbedded  Markov  chain. 
State  t  is  positive  recurrent  in  the  imbedded  Markov  chain  (contained 
in  ergodic  subchain  E^)  and,  for  some  constant  c,  £  c  <  ■»,  Vi, 

j  €  E^,  =»  l  is  positive  recurrent  in  the  SMP.  An  SMP  is  positive 

recurrent  if  all  the  states  in  I+  are  positive  recurrent  in  the  SMP. 

We  remark  that,  if  Z(0)  “  i  and  i  belongs  to  the  same  positive 
recurrent  ergodic  subchain  as  j ,  t  *N^(t)  **  1  / ^  w.p.l,  a  strong  law 

that  follows  immediately  from  renewal  theory.  Under  these  conditions 

(2) 

and  assuming  <  «,  N^(t)  is  asymptotically  normally  distributed 

■3  (2)  2 

with  mean  an<*  variance  *  (j>  j  j  ^  *  a  conse9uence  of  a 

renewal- theoretic  result  found,  for  example,  in  Feller  [16,  p.  359]. 
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VI.  STATIOKARY  PROBABILITIES 


It  is  Important  to  distinguish  between  the  stationary  proba¬ 
bilities  with  respect  to  the  imbedded  Markov  chain  and  the 

stationary  probabilities  {p^}  with  respect  to  the  SMP.^  Thus 
is  the  steady  state  probability  that  the  last  distinguished  state 
entered  is  i.  Hence  the  are  of  direct  interest  in  applications, 

while  the  {tt^}  are  computed  only  as  an  intermediate  step.  We  consider 
first  the  case  I+  finite  and  <  ®,  Vi  €  I+. 

Vj/Hjj.  j  €  Ek,  Z(0)  €  \ 

j  €  Ek,  Z(0)  -  i  6  T 

0.  j  €  T 

0,  j  €  Z(0)  e  E£.  k  *  l 

where  Gtj(«0  was  computed  already  and 


v*jj 


i€E. 


"ivi 


with  here  being  the  stationary  probabilities  for  the  imbedded 

Markov  chain  given  that  Z(0)  €  E^. 

In  the  remainder  of  this  section  we  assume  that  the  imbedded 
Markov  chain  is  irreducible  and  that  the  SMP  is  positive  recurrent, 
where  I+  may  be  finite  or  infinite.  We  also  assume  that  the  mean 


J.n  general,  the  stationary  probabilities  must  be  interpreted 
as  Cesaro  limits.  If  the  process  is  aperiodic,  these  reduce  to 
ordinary  limits. 
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transitlon  times  are  uniformly  bounded  away  from  zero;  i.e.,  0  <  e  < 
v. .  <oo.  With  these  assumptions,  Fabens  [151  shows  that 


Evi 


in  agreement  with  results  given  above  for  the  1+  finite  case.  Define  a(x) 
as  the  time  of  the  last  transition  completion  before  or  at  x  and  t(x)  as 
the  time  of  the  next  transiton  completion  after  x.  The  random  variables 


Y(x)  “  t(x)  -  x  (excess  r.v.) 
6(x)  •*  x  -  o(x)  (shortage  r.v.) 


are  of  interest.  Adding  to  the  previous  assumptions  the  hypotheses 
that  the  mean  recurrence  times  are  finite  and  that  Z(*)  is 

aperiodic,  Fabens  shows  that 

lim  Pf  6(t)  £  x  |z(t)  -  i}  *=  lim  P(Y(t)  s  x|Z(t)  -  i} 

t-»®  t-«D 


[1  -  Hi (u) ]  du. 


This  generalizes  the  well-known  result  from  renewal  theory  for  the 
one  state  case,  obtained  there  as  a  corollary  to  the  key  renewal 
theorem;  see,  e.g.,  Barlow  and  Proschan  [1]. 

The  general  question  of  existence  and  uniqueness  of  stationary 
measures  is  dealt  with  in  Pyke  and  Schaufele  T47].  Cheong  [3]  gives 
conditions  under  which  convergence  to  the  steady  state  is  geometric; 
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see  also  Teugels  [51].  An  estimate  of  the  convergence  rate  is  important, 
li  it  is  high  enough,  troublesome  transient  phenomena  can  be  neglected. 

We  then  pass  directly  to  a  relatively  simple  steady  state  analysis. 
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VII.  EXAMPLE:  THE  M/G/l  QUEUE 

To  illustrate  the  notion  of  stationary  probabilities  for  an  SKP, 

Q 

we  consider  the  M/G/l  queue.  Let 

=  the  stationary  probability  that  i  customers  are  in 
the  system  Just  after  a  random  service  completion 
epoch 

=  the  stationary  probability  that  i  customers  are  in 
the  system  just  after  the  service  completion  epoch 
preceding  a  random  point  in  time 

p.  *  the  stationary  probability  that  i  customers  are  in 
the  system  at  a  random  point  in  time. 

We  assume  that  the  traffic  intensity  Xb  Is  less  than  one,  where  X  is 

the  arrival  rate  and  b  the  mean  service  time,  assumed  positive. 

Although  it  is  easily  shown  that 

Xbr^,  i  i  1 

(1  +  Xb)rro  -  1  -  (Xb)2,  i  -  0, 

it  turns  out  that  p^  ■  tt^ ,  Vi,  a  remarkable  result  originally  due  to 
Khintchlne  [31]  and  derived  in  a  more  elementary  manner  by  Fox  and 
Miller  [24T  using  SMP  theory.  In  bulk  queues  (Fabens  fl5]),  for 
example,  the  stationary  measures  for  the  imbedded  Markov  chain  and 
the  original  queueing  process  are  different. 

Readers  familiar  with  queueing  theory  may  prefer  to  skip  to  the 
last  paragraph  of  this  section.  In  between,  the  standard  manipulations 
yielding  G^(r) ,  the  generating  function  of  the  are  performed. 

g 

The  stationary  probabilities  for  the  imbedded  chain  and  the  SMP 
are  and  fp^,  respectively. 
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Recalling  that  .state  n  means  that  there  are  n  people  in  the 
system  and  a  service  has  just  been  completed,  we  obtain  the  well 
known  transition  matrix  ior  the  imbedded  Markov  chain: 


P  * 


where  the  probability  that  n  customers  arrive  wnile  a  customer  is  being 
serviced  is 

Vc  *=  f  ~T"  <Xt)n  dB(t) 
n  I  n  •  . 

u 

and  B  is  the  service  distribution.  By  the  usual  straightforward 
manipulations,  we  find  that  the  generating  function  of  the  {kR}  is 

Gy(z)  “]Ckizi  "  "  z>)> 

i 

where  g  is  the  Laplace-Stielt jes  transform  of  B,  i.e., 
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To  obtain  the  stationary  vector  rr  for  the  chain,  we  multiply  the  i-th 
relation  determined  by  ttP  =  tt  by  z^  and  sum,  define  the  generating 
funct ion 

gtt(z)  =  E  vl* 

i 

and  obtain  from  the  special  form  of  P  for  this  chain  by  an  easy  calcu¬ 
lation  the  standard  result 


V"> 


TT0(i  -  2)Ck(z) 

Gk(z)  -  z 


Using  the  fact  that  lim  G  (z)  =  1  (i.e.  ,  the  probabilities  sum  to  1) 


-  TT 


z—1 

and  applying  L' Hospital's  rule. 


1  -  Xb. 


Summarizing  our  results  so  far. 


ttq(1  -  z)g(X(l  -  z)) 
P(X(1  -  z))  -  z 


Thus  the  mean  number  in  the  system  averaged  over  service  completion 
2 

epochs  is,  with  a  the  variance  of  the  service  times. 


G"(z)  -  2G'(z)(C'(z)  -  1) 

lim  G  (z)  *  tt  lim  - j - 

z -r  "  Z-1‘  |_  2(G^(z)  -  1  r 


2  2 

(Xbr  +  a 

1  -  Xb 


+  Xb, 


and  by  the  fact  that  rr  •  Pj^,  Vi,  is  also  the  mean  number  in  the 
system  at  random  point  in  time  (in  the  steady  state).  Higher  moments 
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and  probabilities  can  be  obtained  from  the  generating  function  by 
appropriate  differentiations,  which,  however,  become  quite  tedious. 

Having  found  G  (z),  the  Laplace-Stielt jes  transform  of  the  stat- 

TT 

ionary  waiting  distribution  (Pol laczek-Khintchine  formula)  for  the  first 
come,  first  served  (FIFO)  discipline  can  easily  be  found.  The  deriva¬ 
tion  depends  on  the  fact  that,  since  the  arrival  process  is  Poisson, 
an  arrival  plays  the  role  of  a  random  observer.  If  an  arrival  finds 
the  system  empty,  the  conditional  wait  in  queue  is  0.  Otherwise  it 
is  governed  by  the  remaining  processing  time  of  the  customer  in  service, 
the  excess  random  variable,  plus  the  service  times  for  the  customers 

(if  any)  already  in  queue.  Noting  that  G  (z)  =  G  (z) ,  the  interested 

P  Tt 

reader  can  readily  derive  a  version  of  the  Pol laczek-Khintchine  formula, 
namely,  rro  +  Vbftfs)^  [Gp<B(s))  '  ttq].  See  Feller  [16,  p.  3921,  for  an 
alternate  elegant  derivation  that  bypasses  the  calculation  of  G^(z). 

A  third  derivation  follows  from  the  fact  that  the  number  of  customers 
in  the  system  just  after  a  departure  is  the  number  of  arrivals  during 
his  total  wait  (queueing  time  plus  service  time);  the  resulting  equa¬ 
tion  is  solved  by  taking  generating  functions  yielding  the  standard 
form  of  the  Laplace-Stielt jes  transform  of  the  stationary  queueing 

delay  distribution  snQ/(s  -  xO  -  g(s))),  t*ie  more  familiar  version  of 

9 

the  Pollaczek-Khintchine  formula.  For  a  fourth  derivation,  where  the 
(superfluous)  assumption  of  an  absolutely  continuous  failure  distribu¬ 
tion  is  tacitly  made,  see  Cox  and  Miller  [9,  pp.  241-242]. 

- 

Comparison  of  the  two  versions  yields  an  interesting  and  surpris¬ 
ing  identity. 


-23- 


VIII.  ASYMPTOTIC  FORM  OF  M 


From  a  previous  section,  we  know  that 


m..(s)  “  8ij  (s)t1  +  mjj(s>l 


r  «jj(s)  i 

5ij(s)[1+r^)J 


Formally  expanding  e  Sx  in  a  Taylor  series  under  the  integral  defining 
g^ . ,  integrating  termwise,  and  performing  the  appropriate  algebraic 
manipulations  yields  for  i,  j  in  the  same  ergodic  subchain 


(2) 


1  ^ij 

“7—  +-LLT-  *  —  + 


ij'  7 


JJ  2Ujj‘  ^JJ 


o(l), 


whence  by  a  Tauberian  argument 


M„<t)  -~ 

‘JJ 


(2) 

hi _  hi 


ijV  V.it  (Cesaro)  2  2 


a  result  that  can  be  obtained  by  analogy  with  renewal  theory  for 
delayed  recurrent  events,  where  the  time  to  the  first  "renewal"  has 
distribution  and  the  spacing  between  subsequent  renewals  has 
distribution  G^.  If  the  SMP  is  aperiodic,  the  Cesaro  limit  reduces 
to  an  ordinary  limit.  It  can  be  shown  that  the  formal  manipulation 
used  to  obtain  the  asymptotic  expansion  of  m^j(s)  is  justified  if 

<  ».  If  I+  is  finite,  <  ®,  Vi  €  I+  •  <  “>.  A  result 
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that  holds  for  all  t  is  Z(0)  “  j  =>M..(t)  2  t/u..  -  1>  which  follows 

JJ 

from  Barlow  and  Proschan  [1,  Theorem  2.5].  We  can  obtain  a  tighter 
inequality  from  [1]  and,  if  has  increasing  failure  rate,  an  upper 
bound  as  well. 
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IX.  FINITE  SMP'S  WITH  COSTS 


Sometimes  the  performance  of  a  system  is  evaluated  by  probabilistic 
criteria,  such  as  average  delay  in  a  queue,  that  serve  as  surrogates  for 
monetary  loss.  It  is  more  appealing  to  deal  directly  with  expected  loss 
as  a  performance  measure. ^  In  this  section  we  indicate  how  to  do  this. 

Often  in  applications,  costs  are  associated  with  the  transitions. 
Measuring  time  from  the  start  of  an  i  -•  j  transition,  let  C„(x|t)  be 
the  cost  incurred  up  to  time  x  given  that  the  transition  length  is  t. 

The  expected  discounted  cost  for  a  transition  starting  from  state  i 
is  then 


Y.(or) 


j 


®  t 

a  J  dV°  / 


-a* 


dxVxl 


o, 


where  a  cost  incurred  at  time  x  is  discounted  by  the  factor  e 
An  elementary  renewal  type  argument  then  shows  that  v^er),  the  total 
expected  discounted  cost  over  an  infinite  horizon  starting  from  state 
i,  satisfies 


v(ar)  »  Y(a)  +  q(cr)v(of), 

where  or  >  0  and  v(or)  and  y(a)  are  the  vectors  with  components  v^(cr) 
and  y(a) ,  respectively.  Thus,  assuming  a  finite  number  of  states, 

v(or)  “  [I  -  q(ar)]"1  Y(®) . 


similar  remark  applies  to  "chance-constrained"  programming. 
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Since  I  -  q(0)  is  singular  and  a  direct  asymptotic  expansion  is  not 
obvious,  it  is  convenient  to  make  use  of  the  relation  between  q  and 
m  given  earlier  to  study  the  behavior  of  v(ot)  as  9  ->  0+.  Following 
Jewell  [27],  we  have 


v(a)  -  [I  +  m(cr)]Y(or)  • 

Making  use  of  the  expansion  of  m(or)  given  in  the  preceding  section, 
we  find  that,  if  i  is  a  recurrent  state,  v^(cy)  has  the  form 

v.  (er)  ■  1.  /«  +  +  o(l) 

and  a  straightforward  argument  in  Fox  [19]  then  shows  that  this  form 
is  valid  for  any  state;  i.e., 

v(«)  ■  i/a  +  w  +  o(l)  , 

where  expressions  for  l,  the  loss  rate  vector,  and  w,  the  bias  term 
vector,  can  be  found  in  Jewell  [28,  29]  and  Fox  [19],  where  appropriate 
conditions  are  given  to  justify  the  expansion.  Substituting  this 
relation  into  v(cr)  •  V(or)  +  q(a)v(cr)  and  equating  the  coefficients 
of  or  *  and  the  constant  terms,  respectively,  yields** 

P  l  •  t 

y  +  Pw  ■  w  +  y 

'  ?  ’uVj 

“  if  i  is  recurrent. 

**This  procedure  can  be  justified  by  a  simple  contradiction 
argument.  Note  that  qij(a)  ■  Pij(l  -  avn)  +  o(tt)  and  that  the 
loss  rate  for  all  states  in  an  ergodic  subchain  is  the  same. 
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These  expressions  can  be  solved  uniquely  for  £,  but  w  is  determined 

only  up  to  an  additive  constant  in  each  ergodic  subchain;  see,  e.g., 

Denardo  and  Fox  [12].  An  interesting  and  intuitive  result  that  follows 

easily  from  the  above  formulas  is  that  the  loss  rate  for  each  state  in 

(k)  ,  (k) 

an  ergodic  subchain  is  the  same  and  equal  to  tt  'yV "  'v ',  where 
(k) 

nv  '  is  the  stationary  vector  for  the  corresponding  submatrix  and  y' 
and  v*  are  the  restrictions  Y  and  v»  respectively,  to  E^.  This  formula 
can  be  rewritten  as 

pi<W’ 

which  is  the  sum  of  the  expected  cost  per  unit  time  in  each  state  of 
weighted  by  the  respective  stationary  probabilities  for  the  SMP. 

The  loss  rates  for  the  transient  states  are  obtained  from  the  fact 
reflected  in  Pi  ■  i  that  the  loss  rate  for  a  state  is  given  by  the 
appropriate  convex  combination  and  that  I  -  A,  where  A  is  the  sub¬ 
matrix  corresponding  to  the  transient  states,  is  invertible. 

Denoting  the  undiscounted  loss  up  to  time  t  by  L(t),  we  obtain 
from  the  asymptotic  expansion  of  v(O)  that 

L(t)  -  ft  - -  *•  w. 

(Cesaro) 

Jewell  [30]  studies  the  fluctuations  in  cumulative  loss  in  what 
is  essentially  the  one-state  case.  If  the  imbedded  Markov  chain  is 
ergodic,  these  results  extend  in  principle  to  n-state  problems  by 
considering  Gj^  and  the  distribution  of  cumulative  loss  until  the 


E 

i€Ek 


j 

V 


first  return.  In  general,  the  calculation  would  be  tedious.  Besides, 
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we  are  distinguishing  here  only  one  state  out  of  n.  Apparently,  no 
one  has  dealt  with  fluctuation  theory  for  the  n-state  case  directly. 

A  related  topic  is  a  central  limit  theorem  for  cumulative  loss. 
Since  cumulative  loss  is  an  example  of  a  functional  of  a  Markov 
Renewal  Process,  results  of  Pyke  and  Schaufele  [46]  apply. 
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X.  MARKOV  RENEWAL  PROGRAMMING 


The  situation  becomes  more  interesting  when,  at  each  state  i, 
one  has  a  set  of  options  and  the  choice  at  i  simultaneously  deter¬ 
mines  p..,  F..,  and  C..  for  all  j  6  1+.  The  goal  is  to  choose 
ij  ij’  ij 

a  policy  that  minimizes  either  the  expected  discounted  loss  or  the 
loss  rate.  In  the  latter  case,  an  appropriate  secondary  objective  is 
to  minimize  (£,  w)  lexicographical ly ,  which  is  especially  important 
when  some  policies  can  have  transient  states.  With  either  criterion, 
an  optimal  policy  can  be  found  by  linear  programming  when  I+  and 
X-^j+A^  are  finite.  For  details,  see,  e.g.,  Jewell  T28,  291,  Fox 
[19,  221,  Denardo  [10,  11],  and  Denardo  and  Fox  [12],  where  references 
to  the  earlier  (extensive)  literature  on  the  subject  are  given.  The 
linear  programming  formulation  facilitates  sensitivity  analyses  and 
parametric  studies.  Controlling  roundoff  errors  is  probably  less 
difficult  in  the  averaging  version;  see  related  remarks  in  [12]. 

Some  papers  treat  the  I+  infinite  case,  but  the  author  believes 
that,  for  applications,  the  general  theory  developed  so  far  for  that 
case  is  inadequate  and  that  particular  problems  are  best  attacked  on 
an  ad  hoc  basis.  The  averaging  version  of  the  infinite  state  case 
apparently  has  been  studied  only  in  the  discrete  time  setup;  see, 
e.g.,  Derman  [14],  Ross  [4b].  However,  in  the  discounted  continuous 
time  version  no  new  theoretical  problems  arise  when  the  problem  is 
approached  via  contraction  mappings  (Denardo  [10]).  For  the  case 
where  I+  is  finite  but  the  finiteness  restriction  on  XA^  is  dropped, 
see  Fox  [201 .  The  connection  with  generalized  linear  programming  and 
column  generators  is  outlined  in  [12], 
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Miller  [32,  33"5  treats  the  continuous  time  Markov  process  case 
with  loss  proportional  to  the  transition  time. 

Markov  renewal  programming  problems  a>-e  a  fertile  source  of  large 
scale  linear  programs.  Many  problems  that  look  deceptively  simple  at 
first  sight  can  lead  to  linear  programs  with  hundreds  or  thousands 
of  constraints  because  of  the  detailed  state  description  required  to 
make  all  decision  points  regeneration  points.  But  often  we  need  not 
throw  in  the  towel.  Another  look  generally  reveals  that  the  con¬ 
straint  matrix  is  sparse  and  structured  so  as  to  be  amenable  to 
decomposition. 
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XI.  ESTIMATION.  INFERENCE,  AND  ADAPTIVE  CONTROL 


(F  J  and  their  large  sample  distributions.  For  statistical  inference 
in  birth  and  death  queueing  modeii,  see  Wolff  [54 "V  Both  of  the  fore¬ 
going  approaches  are  objectivist,  i.e.,  non-Bayesian.  When  a  large 
number  of  observations  are  at  hand,  the  objectivist  approach  is  un¬ 
objectionable  and  difficulties  stemming  from  a  possible  lack  of  con¬ 
sensus  of  prior  belief  do  not  emerge.  On  the  other  hand,  when  the 

observations  are  few  or  nonexistent,  as  is  common,  a  Bayesian  approach 

12 

incorporating  prior  beliefs  and  loss  functions  is  essential.  Such 
an  approach  may  be  formal  or  may  simply  consist  of  a  sensitivity  analy¬ 
sis  with  the  outcomes  being  given  subjective  weights.  In  the  realm  of 
decision  making,  policies  should  adapt  to  modified  beliefs  as  more 
observations  are  taken. 

Unfortunately,  when  the  tradeoff  between  information  acquisition 
and  immediate  losses  is  explicitly  included  in  the  problem  formulation, 
the  number  of  states  generally  explodes.  Generally  explicit  inclusion 
is  advisable,  because  if  an  average  cost  criterion  is  interpreted 
literally,  policies  that  are  absurd  for  any  positive  discount  rate  can 
result.  Thus,  from  a  practical  viewpoint,  for  relatively  few  problems 
(see,  e.g.,  T21D  can  "optimal"  adaptive  policies  be  found;  for  the 
remainder,  it  appears  that  we  must  be  content  with  heuristic  devices. 
This  area  remains  largely  unexplored  and  is  ripe  for  investigation. 

12 

This  is,  of  course,  a  statement  of  the  author's  opinion.  These 
matters  are  highly  controversial. 
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